Skip to content

Conversation

amolsr
Copy link
Contributor

@amolsr amolsr commented Oct 11, 2025

What's Changed

  • Rewrote docs/integrations/data-ingestion/kafka/kafka-clickhouse-connect-sink.md with detailed coverage of:
  • When performance tuning is actually needed (and when it's not)
  • Detailed explanation of data flow through the connector
  • Batch size configuration with recommended settings and trade-offs
  • Complete section on asynchronous inserts including use cases, configuration, and important caveats about exactly-once semantics
  • Connector parallelism and scaling strategies
  • ClickHouse table engine considerations and settings
  • Connection pooling and timeout configuration
  • Monitoring key metrics and troubleshooting common issues
  • Best practices summary
  • Production-ready high-throughput configuration example

Closes ClickHouse/clickhouse-kafka-connect#595

@amolsr amolsr requested review from a team as code owners October 11, 2025 22:06
Copy link

vercel bot commented Oct 11, 2025

@amolsr is attempting to deploy a commit to the ClickHouse Team on Vercel.

A member of the Team first needs to authorize it.

@amolsr
Copy link
Contributor Author

amolsr commented Oct 16, 2025

Can we do a quick review on this? The CI checks have passed, so I think it’s ready to be merged.

Copy link

vercel bot commented Oct 16, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
clickhouse-docs Ready Ready Preview Oct 16, 2025 10:17am

@amolsr
Copy link
Contributor Author

amolsr commented Oct 17, 2025

Hi @mshustov can you review this?

- **Small message sizes**: When dealing with many small messages that would benefit from server-side batching

This fetched data is then consumed by the connector client polling for messages, where the amount for each poll is controlled by `max.poll.records` - please note that fetch is independent of poll, though!
Performance tuning is **NOT typically needed** when:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good section! I agree that any optimization should start with current state assessment and checking if problem exists.

@Blargian Blargian merged commit 62c7c3d into ClickHouse:main Oct 17, 2025
10 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DOC] Throughput and async insert

3 participants